{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wide and Deep on TensorFlow (notebook style)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2016 Google Inc. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
    " http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This notebook uses the tf.learn API in TensorFlow to answer a yes/no question. This is called a binary classification problem: Given census data about a person such as age, gender, education and occupation (the features), we will try to predict whether or not the person earns more than 50,000 dollars a year (the target label). \n",
    "\n",
    "Given an individual's information our model will output a number between 0 and 1, which can be interpreted as the model's certainty that the individual has an annual income of over 50,000 dollars, (1=True, 0=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports and constants\n",
    "First we'll import our libraries and set up some strings for column names. We also print out the version of TensorFlow we are running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow version 1.13.0-rc2\n",
      "\n",
      "Feature columns are:  ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13', 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'] \n",
      "\n",
      "Columns and data as a dict:  {'I1': 0, 'I2': 127, 'I3': 1, 'I4': 3, 'I5': 1683, 'I6': 19, 'I7': 26, 'I8': 17, 'I9': 475, 'I10': 0, 'I11': 9, 'I12': 0, 'I13': 3, 'C1': '05db9164', 'C2': '8947f767', 'C3': '11c9d79e', 'C4': '52a787c8', 'C5': '4cf72387', 'C6': 'fbad5c96', 'C7': '18671b18', 'C8': '0b153874', 'C9': 'a73ee510', 'C10': 'ceb10289', 'C11': '77212bd7', 'C12': '79507c6b', 'C13': '7203f04e', 'C14': '07d13a8f', 'C15': '2c14c412', 'C16': '49013ffe', 'C17': '8efede7f', 'C18': 'bd17c3da', 'C19': 'f6a3e43b', 'C20': 'a458ea53', 'C21': '35cd95c9', 'C22': 'ad3062eb', 'C23': 'c7dc6720', 'C24': '3fdb382b', 'C25': '010f6491', 'C26': '49d68486'} \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO) # Set to INFO for tracking training, default is WARN. ERROR for least messages\n",
    "\n",
    "print(\"Using TensorFlow version %s\\n\" % (tf.__version__))\n",
    "\n",
    "\n",
    "CONTINUOUS_COLUMNS =  [\"I\"+str(i) for i in range(1,14)] # 1-13 inclusive\n",
    "CATEGORICAL_COLUMNS = [\"C\"+str(i) for i in range(1,27)] # 1-26 inclusive\n",
    "LABEL_COLUMN = [\"clicked\"]\n",
    "\n",
    "TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "# TEST_DATA_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "FEATURE_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "print('Feature columns are: ', FEATURE_COLUMNS, '\\n')\n",
    "\n",
    " # label is 1\n",
    "sample = [ 0 , 2, 11, 5, 10262, 34, 2, 4, 5,0 , 1,0 , 5, \"be589b51\", \"287130e0\", \"cd7a7a22\", \"fb7334df\", \"25c83c98\",\"0\" , \"6cdb3998\", \"361384ce\", \"a73ee510\", \"3ff10fb2\", \"5874c9c9\", \"976cbd4c\", \"740c210d\", \"1adce6ef\", \"310d155b\", \"07eb8110\", \"07c540c4\", \"891589e7\", \"18259a83\", \"a458ea53\", \"a0ab60ca\",\"0\" , \"32c7478e\", \"a052b1ed\", \"9b3e8820\", \"8967c0d2\"]\n",
    "\n",
    "# label is 1\n",
    "sample = [ 0, 127, 1, 3, 1683, 19, 26, 17, 475, 0, 9, 0, 3, \"05db9164\", \"8947f767\", \"11c9d79e\", \"52a787c8\", \"4cf72387\", \"fbad5c96\", \"18671b18\", \"0b153874\", \"a73ee510\", \"ceb10289\", \"77212bd7\", \"79507c6b\", \"7203f04e\", \"07d13a8f\", \"2c14c412\", \"49013ffe\", \"8efede7f\", \"bd17c3da\", \"f6a3e43b\", \"a458ea53\", \"35cd95c9\", \"ad3062eb\", \"c7dc6720\", \"3fdb382b\", \"010f6491\", \"49d68486\"]\n",
    "\n",
    "print('Columns and data as a dict: ', dict(zip(FEATURE_COLUMNS, sample)), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input file parsing\n",
    "\n",
    "This section puts the file into a `Reader` which reads from the file one batch at a time. \n",
    "\n",
    "We set up the Tensors to be a dictionary of features mapping from their string name to the tensor value.\n",
    "\n",
    "Note that the `_input_fn()` function is wrapped, enabling it to be used for different files.\n",
    "\n",
    "NOTE: This reads from the input file directly via TensorFlow, rather than using an intermediate tool such as pandas to load the entire dataset into memory first. This is done to enable the system to scale to large inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about input functions\n",
    "\n",
    "The input function is how we will feed the input data into the model during training and evaluation. \n",
    "The structure that must be returned is a pair, where the first element is a dict of the column names (features) mapped to a tensor of values, and the 2nd element is a tensor of values representing the answers (labels). Recall that a tensor is just a general term for an n-dimensional array.\n",
    "\n",
    "This could be represented as: `map(column_name => [Tensor of values]) , [Tensor of labels])`\n",
    "\n",
    "More concretely, for this particular dataset, something like this:\n",
    "\n",
    "    { \n",
    "      'age':            [ 39, 50, 38, 53, 28, … ], \n",
    "      'marital_status': [ 'Married-civ-spouse', 'Never-married', 'Widowed', 'Widowed' … ],\n",
    "       ...\n",
    "      'gender':           ['Male', 'Female', 'Male', 'Male', 'Female',, … ], \n",
    "    } , \n",
    "    [ 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1]\n",
    "    \n",
    "Additionally, we define which columns of the input data we will treat as categorical vs continuous, using the global `CATEGORICAL_COLUMNS`.\n",
    "\n",
    "You can try different values for `BATCH_SIZE` to see how they impact your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High-level structure of input functions for CSV-style data\n",
    "1. Queue file(s)\n",
    "2. Read a batch of data from the next file\n",
    "3. Create record defaults, generally 0 for continuous values, and \"\" for categorical. You can use named types if you prefer\n",
    "4. Decode the CSV and restructure it to be appropriate for the graph's input format\n",
    "    * `zip()` column headers with the data\n",
    "    * `pop()` off the label column(s)\n",
    "    * Remove/pop any unneeded column(s)\n",
    "    * Run `tf.expand_dims()` on categorical columns\n",
    "    5. Return the pair: `(feature_dict, label_array)`\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input function configured\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 400\n",
    "\n",
    "def generate_input_fn(filename, batch_size=BATCH_SIZE):\n",
    "    def _input_fn():\n",
    "        filename_queue = tf.train.string_input_producer([filename])\n",
    "        reader = tf.TextLineReader()\n",
    "        # Reads out batch_size number of lines\n",
    "        key, value = reader.read_up_to(filename_queue, num_records=batch_size)\n",
    "        \n",
    "        # 1 int label, 13 ints, 26 strings\n",
    "        cont_defaults = [ [0] for i in range(1,14) ]\n",
    "        cate_defaults = [ [\" \"] for i in range(1,27) ]\n",
    "        label_defaults = [ [0] ]\n",
    "        column_headers = TRAIN_DATA_COLUMNS\n",
    "        # The label is the first column of the data.\n",
    "        record_defaults = label_defaults + cont_defaults + cate_defaults\n",
    "        \n",
    "        # Decode CSV data that was just read out. \n",
    "        # Note that this does NOT return a dict, \n",
    "        # so we will need to zip it up with our headers\n",
    "        columns = tf.decode_csv(\n",
    "            value, record_defaults=record_defaults)\n",
    "        \n",
    "        # all_columns is a dictionary that maps from column names to tensors of the data.\n",
    "        all_columns = dict(zip(column_headers, columns))\n",
    "        \n",
    "        # Pop and save our labels \n",
    "        # dict.pop() returns the popped array of values; exactly what we need!\n",
    "        labels = all_columns.pop(LABEL_COLUMN[0])\n",
    "        \n",
    "        # the remaining columns are our features\n",
    "        features = all_columns \n",
    "\n",
    "        # Sparse categorical features must be represented with an additional dimension. \n",
    "        # There is no additional work needed for the Continuous columns; they are the unaltered columns.\n",
    "        # See docs for tf.SparseTensor for more info\n",
    "        for feature_name in CATEGORICAL_COLUMNS:\n",
    "            features[feature_name] = tf.expand_dims(features[feature_name], -1)\n",
    "\n",
    "        return features, labels\n",
    "\n",
    "    return _input_fn\n",
    "\n",
    "print('input function configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Feature Columns\n",
    "This section configures the model with the information about the model. There are many parameters here to experiment with to see how they affect the accuracy.\n",
    "\n",
    "This is the bulk of the time and energy that is often spent on making a machine learning model work, called *feature selection* or *feature engineering*. We choose the features (columns) we will use for training, and apply any additional transformations to them as needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sparse Columns\n",
    "First we build the sparse columns.\n",
    "\n",
    "Use `sparse_column_with_keys()` for columns that we know all possible values for.\n",
    "\n",
    "Use `sparse_column_with_hash_bucket()` for columns that we want the the library to automatically map values for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "Wide/Sparse columns configured [_SparseColumnHashed(column_name='C1', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C2', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C3', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C4', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C5', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C6', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C7', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C8', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C9', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C10', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C11', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C12', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C13', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C14', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C15', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C16', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C17', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C18', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C19', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C20', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C21', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C22', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C23', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C24', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C25', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C26', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string)]\n"
     ]
    }
   ],
   "source": [
    "# Sparse base columns.\n",
    "# C1 = tf.contrib.layers.sparse_column_with_hash_bucket('C1', hash_bucket_size=1000)\n",
    "# C2 = tf.contrib.layers.sparse_column_with_hash_bucket('C2', hash_bucket_size=1000)\n",
    "# C3 = tf.contrib.layers.sparse_column_with_hash_bucket('C3', hash_bucket_size=1000)\n",
    "# ...\n",
    "# Cn = tf.contrib.layers.sparse_column_with_hash_bucket('Cn', hash_bucket_size=1000)\n",
    "# wide_columns = [C1, C2, C3, ... , Cn]\n",
    "\n",
    "wide_columns = []\n",
    "for name in CATEGORICAL_COLUMNS:\n",
    "    wide_columns.append(tf.contrib.layers.sparse_column_with_hash_bucket(\n",
    "            name, hash_bucket_size=1000))\n",
    "\n",
    "print('Wide/Sparse columns configured', wide_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous columns\n",
    "Second, configure the real-valued columns using `real_valued_column()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deep/continuous columns configured [_RealValuedColumn(column_name='I1', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I2', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I3', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I4', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I5', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I6', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I7', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I8', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I9', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I10', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I11', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I12', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I13', dimension=1, default_value=None, dtype=tf.float32, normalizer=None)]\n"
     ]
    }
   ],
   "source": [
    "# Continuous base columns.\n",
    "# I1 = tf.contrib.layers.real_valued_column(\"I1\")\n",
    "# I2 = tf.contrib.layers.real_valued_column(\"I2\")\n",
    "# I3 = tf.contrib.layers.real_valued_column(\"I3\")\n",
    "# ...\n",
    "# In = tf.contrib.layers.real_valued_column(\"In\")\n",
    "# deep_columns = [I1, I2, I3, ... , In]\n",
    "\n",
    "deep_columns = []\n",
    "for name in CONTINUOUS_COLUMNS:\n",
    "    deep_columns.append(tf.contrib.layers.real_valued_column(name))\n",
    "\n",
    "print('deep/continuous columns configured', deep_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformations\n",
    "Now for the interesting stuff. We will employ a couple of techniques to get even more out of the data.\n",
    " \n",
    "* **bucketizing** turns what would have otherwise been a continuous feature into a categorical one. \n",
    "* **feature crossing** allows us to compute a model weight for specific pairings across columns, rather than learning them as independently. This essentially encodes related columns together, for situations where having 2 (or more) columns being certain values is meaningful. \n",
    "\n",
    "Only categorical features can be crossed. This is one reason why age has been bucketized.\n",
    "\n",
    "For example, crossing education and occupation would enable the model to learn about: \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Exec-managerial\"\n",
    "\n",
    "or perhaps \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Craft-repair\"\n",
    "\n",
    "We do a few combined features (feature crosses) here. \n",
    "\n",
    "Add your own, based on your intuitions about the dataset, to try to improve on the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transformations complete\n"
     ]
    }
   ],
   "source": [
    "# No known Transformations. Can add some if desired. \n",
    "# Examples from other datasets are shown below.\n",
    "\n",
    "# age_buckets = tf.contrib.layers.bucketized_column(age,\n",
    "#             boundaries=[ 18, 25, 30, 35, 40, 45, 50, 55, 60, 65 ])\n",
    "# education_occupation = tf.contrib.layers.crossed_column([education, occupation], \n",
    "#                                                         hash_bucket_size=int(1e4))\n",
    "# age_race_occupation = tf.contrib.layers.crossed_column([age_buckets, race, occupation], \n",
    "#                                                        hash_bucket_size=int(1e6))\n",
    "# country_occupation = tf.contrib.layers.crossed_column([native_country, occupation], \n",
    "#                                                       hash_bucket_size=int(1e4))\n",
    "\n",
    "print('Transformations complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group feature columns into 2 objects\n",
    "\n",
    "The wide columns are the sparse, categorical columns that we specified, as well as our hashed, bucket, and feature crossed columns. \n",
    "\n",
    "The deep columns are composed of embedded categorical columns along with the continuous real-valued columns. **Column embeddings** transform a sparse, categorical tensor into a low-dimensional and dense real-valued vector. The embedding values are also trained along with the rest of the model. For more information about embeddings, see the TensorFlow tutorial on [Vector Representations Words](https://www.tensorflow.org/tutorials/word2vec/), or [Word Embedding](https://en.wikipedia.org/wiki/Word_embedding) on Wikipedia.\n",
    "\n",
    "The higher the dimension of the embedding is, the more degrees of freedom the model will have to learn the representations of the features. We are starting with an 8-dimension embedding for simplicity, but later you can come back and increase the dimensionality if you wish.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "WARNING:tensorflow:The default stddev value of initializer was changed from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" in core implementation (tf.feature_column.embedding_column).\n",
      "wide and deep columns configured\n"
     ]
    }
   ],
   "source": [
    "# Wide columns and deep columns.\n",
    "# wide_columns = [gender, race, native_country,\n",
    "#       education, occupation, workclass,\n",
    "#       marital_status, relationship,\n",
    "#       age_buckets, education_occupation,\n",
    "#       age_race_occupation, country_occupation]\n",
    "\n",
    "# deep_columns = [\n",
    "#   tf.contrib.layers.embedding_column(workclass, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(education, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(marital_status, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(gender, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(relationship, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(race, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(native_country, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(occupation, dimension=8),\n",
    "#   age,\n",
    "#   education_num,\n",
    "#   capital_gain,\n",
    "#   capital_loss,\n",
    "#   hours_per_week,\n",
    "# ]\n",
    "\n",
    "# Embeddings for wide columns into deep columns\n",
    "for col in wide_columns:\n",
    "    deep_columns.append(tf.contrib.layers.embedding_column(col, \n",
    "                                                           dimension=8))\n",
    "\n",
    "print('wide and deep columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the model\n",
    "\n",
    "You can train either a \"wide\" model, a \"deep\" model, or a \"wide and deep\" model, using the classifiers below. Try each one and see what kind of results you get.\n",
    "\n",
    "* **Wide**: Linear Classifier\n",
    "* **Deep**: Deep Neural Net Classifier\n",
    "* **Wide & Deep**: Combined Linear and Deep Classifier\n",
    "\n",
    "The `hidden_units` or `dnn_hidden_units` argument is to specify the size of each layer of the deep portion of the network. For example, `[12, 20, 15]` would create a network with the first layer of size 12, the second layer of size 20, and a third layer of size 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory = models/model_WIDE_AND_DEEP_1563712821\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1263b9da0>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1563712821'}\n",
      "estimator built\n"
     ]
    }
   ],
   "source": [
    "def create_model_dir(model_type):\n",
    "    # Returns something like models/model_WIDE_AND_DEEP_1493043407\n",
    "    return 'models/model_' + model_type + '_' + str(int(time.time()))\n",
    "\n",
    "# Specify the desired model_dir \n",
    "def get_model(model_type, model_dir):\n",
    "    print(\"Model directory = %s\" % model_dir)\n",
    "    \n",
    "    # There are more options here than shown here. \n",
    "    # We are using this to show additional checkpointing for illustrative purposes.\n",
    "    # In a real system with far more samples, you would \n",
    "    #     likely choose to save checkpoints less frequently.\n",
    "    runconfig = tf.contrib.learn.RunConfig(\n",
    "        save_checkpoints_secs=None,\n",
    "        save_checkpoints_steps = 100,\n",
    "    )\n",
    "    \n",
    "    m = None\n",
    "    \n",
    "    # Linear Classifier\n",
    "    if model_type == 'WIDE':\n",
    "        m = tf.contrib.learn.LinearClassifier(\n",
    "            model_dir=model_dir, \n",
    "            feature_columns=wide_columns)\n",
    "\n",
    "    # Deep Neural Net Classifier\n",
    "    if model_type == 'DEEP':\n",
    "        m = tf.contrib.learn.DNNClassifier(\n",
    "            model_dir=model_dir,\n",
    "            feature_columns=deep_columns,\n",
    "            hidden_units=[100, 50, 25])\n",
    "\n",
    "    # Combined Linear and Deep Classifier\n",
    "    if model_type == 'WIDE_AND_DEEP':\n",
    "        m = tf.contrib.learn.DNNLinearCombinedClassifier(\n",
    "            model_dir=model_dir,\n",
    "            linear_feature_columns=wide_columns,\n",
    "            dnn_feature_columns=deep_columns,\n",
    "            dnn_hidden_units=[100, 70, 50, 25],\n",
    "            config=runconfig)\n",
    "        \n",
    "    print('estimator built')\n",
    "    \n",
    "    return m\n",
    "    \n",
    "\n",
    "MODEL_TYPE = 'WIDE_AND_DEEP'\n",
    "model_dir = create_model_dir(model_type=MODEL_TYPE)\n",
    "m = get_model(model_type=MODEL_TYPE, model_dir=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Showing that canned estimators return an instance of 'Evaluable'\n",
    "\n",
    "from tensorflow.contrib.learn.python.learn import evaluable\n",
    "isinstance(m, evaluable.Evaluable)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit the model (train it)\n",
    "\n",
    "Run `fit()` to train the model. You can experiment with the `train_steps` and `BATCH_SIZE` parameters.\n",
    "\n",
    "This can take some time, depending on the values chosen for `train_steps` and `BATCH_SIZE`.\n",
    "\n",
    "Our datafile is hosted on Google Cloud Storage; the reader we created at the beginning knows how to read from it.\n",
    "\n",
    "If you don't want to download a new copy of the dataset each time your script runs, you can download it locally using \n",
    "\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/train.csv .\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv .\n",
    "    \n",
    "If you want to download it manually, use\n",
    "\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/eval.csv\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the cloud or local depending on your preference\n",
    "\n",
    "# CLOUD\n",
    "#train_file = \"gs://dataset-uploader/criteo-kaggle/medium_version/train.csv\"\n",
    "#eval_file  = \"gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv\"\n",
    "\n",
    "# LOCAL. Update these paths as appropriate\n",
    "train_file = \"data/train.csv\"\n",
    "eval_file  = \"data/eval.csv\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563712821/model.ckpt-2002\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1070: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file utilities to get mtimes.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 2002 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.46591255, step = 2004\n",
      "INFO:tensorflow:Saving checkpoints for 2104 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 17.9533\n",
      "INFO:tensorflow:loss = 0.4239883, step = 2204 (9.021 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2206 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 24.1395\n",
      "INFO:tensorflow:Saving checkpoints for 2308 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 64.2382\n",
      "INFO:tensorflow:loss = 0.5071357, step = 2404 (3.049 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2410 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 73.7756\n",
      "INFO:tensorflow:Saving checkpoints for 2512 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 82.7233\n",
      "INFO:tensorflow:loss = 0.44974786, step = 2604 (2.522 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2614 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 84.5963\n",
      "INFO:tensorflow:Saving checkpoints for 2716 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 85.6908\n",
      "INFO:tensorflow:loss = 0.5089059, step = 2804 (2.390 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2818 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 81.389\n",
      "INFO:tensorflow:Saving checkpoints for 2920 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 85.8974\n",
      "INFO:tensorflow:loss = 0.5567602, step = 3004 (2.440 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3022 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 86.5788\n",
      "INFO:tensorflow:Saving checkpoints for 3124 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 91.2749\n",
      "INFO:tensorflow:loss = 0.45711896, step = 3204 (2.295 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3226 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 78.8252\n",
      "INFO:tensorflow:Saving checkpoints for 3328 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 86.447\n",
      "INFO:tensorflow:loss = 0.4888687, step = 3404 (2.420 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3430 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 90.0187\n",
      "INFO:tensorflow:Saving checkpoints for 3532 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 90.6416\n",
      "INFO:tensorflow:loss = 0.5528298, step = 3604 (2.214 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3634 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 89.9814\n",
      "INFO:tensorflow:Saving checkpoints for 3736 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 87.1527\n",
      "INFO:tensorflow:loss = 0.48065114, step = 3804 (2.288 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3838 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 88.0496\n",
      "INFO:tensorflow:Saving checkpoints for 3940 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 78.4539\n",
      "INFO:tensorflow:loss = 0.4887472, step = 4004 (2.479 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 4004 into models/model_WIDE_AND_DEEP_1563712821/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.4887472.\n",
      "fit done\n",
      "CPU times: user 57.8 s, sys: 3.87 s, total: 1min 1s\n",
      "Wall time: 47.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# This can be found with\n",
    "# wc -l train.csv\n",
    "train_sample_size = 800000\n",
    "train_steps = train_sample_size/BATCH_SIZE # 8000/40 = 200\n",
    "\n",
    "m.fit(input_fn=generate_input_fn(train_file, BATCH_SIZE), steps=train_steps)\n",
    "\n",
    "print('fit done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate the accuracy of the model\n",
    "Let's see how the model did. We will evaluate all the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T12:54:55Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563712821/model.ckpt-4004\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [50/500]\n",
      "INFO:tensorflow:Evaluation [100/500]\n",
      "INFO:tensorflow:Evaluation [150/500]\n",
      "INFO:tensorflow:Evaluation [200/500]\n",
      "INFO:tensorflow:Evaluation [250/500]\n",
      "INFO:tensorflow:Evaluation [300/500]\n",
      "INFO:tensorflow:Evaluation [350/500]\n",
      "INFO:tensorflow:Evaluation [400/500]\n",
      "INFO:tensorflow:Evaluation [450/500]\n",
      "INFO:tensorflow:Evaluation [500/500]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-12:55:02\n",
      "INFO:tensorflow:Saving dict for global step 4004: accuracy = 0.768895, accuracy/baseline_label_mean = 0.251165, accuracy/threshold_0.500000_mean = 0.768895, auc = 0.73178864, auc_precision_recall = 0.48892838, global_step = 4004, labels/actual_label_mean = 0.251165, labels/prediction_mean = 0.25311968, loss = 0.49670908, precision/positive_threshold_0.500000_mean = 0.6351024, recall/positive_threshold_0.500000_mean = 0.1877252\n",
      "evaluate done\n",
      "Accuracy: 0.768895\n",
      "{'loss': 0.49670908, 'accuracy': 0.768895, 'labels/prediction_mean': 0.25311968, 'labels/actual_label_mean': 0.251165, 'accuracy/baseline_label_mean': 0.251165, 'auc': 0.73178864, 'auc_precision_recall': 0.48892838, 'accuracy/threshold_0.500000_mean': 0.768895, 'precision/positive_threshold_0.500000_mean': 0.6351024, 'recall/positive_threshold_0.500000_mean': 0.1877252, 'global_step': 4004}\n",
      "CPU times: user 14.3 s, sys: 863 ms, total: 15.2 s\n",
      "Wall time: 10.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "eval_sample_size = 200000 # this can be found with a 'wc -l eval.csv'\n",
    "eval_steps = eval_sample_size/BATCH_SIZE # 2000/40 = 50\n",
    "\n",
    "results = m.evaluate(input_fn=generate_input_fn(eval_file), \n",
    "                     steps=eval_steps)\n",
    "print('evaluate done')\n",
    "\n",
    "print('Accuracy: %s' % results['accuracy'])\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'I1': <tf.Tensor 'ExpandDims_26:0' shape=(1,) dtype=int32>, 'I2': <tf.Tensor 'ExpandDims_27:0' shape=(1,) dtype=int32>, 'I3': <tf.Tensor 'ExpandDims_28:0' shape=(1,) dtype=int32>, 'I4': <tf.Tensor 'ExpandDims_29:0' shape=(1,) dtype=int32>, 'I5': <tf.Tensor 'ExpandDims_30:0' shape=(1,) dtype=int32>, 'I6': <tf.Tensor 'ExpandDims_31:0' shape=(1,) dtype=int32>, 'I7': <tf.Tensor 'ExpandDims_32:0' shape=(1,) dtype=int32>, 'I8': <tf.Tensor 'ExpandDims_33:0' shape=(1,) dtype=int32>, 'I9': <tf.Tensor 'ExpandDims_34:0' shape=(1,) dtype=int32>, 'I10': <tf.Tensor 'ExpandDims_35:0' shape=(1,) dtype=int32>, 'I11': <tf.Tensor 'ExpandDims_36:0' shape=(1,) dtype=int32>, 'I12': <tf.Tensor 'ExpandDims_37:0' shape=(1,) dtype=int32>, 'I13': <tf.Tensor 'ExpandDims_38:0' shape=(1,) dtype=int32>, 'C1': <tf.Tensor 'ExpandDims:0' shape=(1,) dtype=string>, 'C2': <tf.Tensor 'ExpandDims_1:0' shape=(1,) dtype=string>, 'C3': <tf.Tensor 'ExpandDims_2:0' shape=(1,) dtype=string>, 'C4': <tf.Tensor 'ExpandDims_3:0' shape=(1,) dtype=string>, 'C5': <tf.Tensor 'ExpandDims_4:0' shape=(1,) dtype=string>, 'C6': <tf.Tensor 'ExpandDims_5:0' shape=(1,) dtype=string>, 'C7': <tf.Tensor 'ExpandDims_6:0' shape=(1,) dtype=string>, 'C8': <tf.Tensor 'ExpandDims_7:0' shape=(1,) dtype=string>, 'C9': <tf.Tensor 'ExpandDims_8:0' shape=(1,) dtype=string>, 'C10': <tf.Tensor 'ExpandDims_9:0' shape=(1,) dtype=string>, 'C11': <tf.Tensor 'ExpandDims_10:0' shape=(1,) dtype=string>, 'C12': <tf.Tensor 'ExpandDims_11:0' shape=(1,) dtype=string>, 'C13': <tf.Tensor 'ExpandDims_12:0' shape=(1,) dtype=string>, 'C14': <tf.Tensor 'ExpandDims_13:0' shape=(1,) dtype=string>, 'C15': <tf.Tensor 'ExpandDims_14:0' shape=(1,) dtype=string>, 'C16': <tf.Tensor 'ExpandDims_15:0' shape=(1,) dtype=string>, 'C17': <tf.Tensor 'ExpandDims_16:0' shape=(1,) dtype=string>, 'C18': <tf.Tensor 'ExpandDims_17:0' shape=(1,) dtype=string>, 'C19': <tf.Tensor 'ExpandDims_18:0' shape=(1,) dtype=string>, 'C20': <tf.Tensor 'ExpandDims_19:0' shape=(1,) dtype=string>, 'C21': <tf.Tensor 'ExpandDims_20:0' shape=(1,) dtype=string>, 'C22': <tf.Tensor 'ExpandDims_21:0' shape=(1,) dtype=string>, 'C23': <tf.Tensor 'ExpandDims_22:0' shape=(1,) dtype=string>, 'C24': <tf.Tensor 'ExpandDims_23:0' shape=(1,) dtype=string>, 'C25': <tf.Tensor 'ExpandDims_24:0' shape=(1,) dtype=string>, 'C26': <tf.Tensor 'ExpandDims_25:0' shape=(1,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563697267/model.ckpt-2002\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<generator object _as_iterable at 0x12a824f48>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def pred_fn():\n",
    "    sample = [ 0, 127, 1, 3, 1683, 19, 26, 17, 475, 0, 9, 0, 3, \"05db9164\", \"8947f767\", \"11c9d79e\", \"52a787c8\", \"4cf72387\", \"fbad5c96\", \"18671b18\", \"0b153874\", \"a73ee510\", \"ceb10289\", \"77212bd7\", \"79507c6b\", \"7203f04e\", \"07d13a8f\", \"2c14c412\", \"49013ffe\", \"8efede7f\", \"bd17c3da\", \"f6a3e43b\", \"a458ea53\", \"35cd95c9\", \"ad3062eb\", \"c7dc6720\", \"3fdb382b\", \"010f6491\", \"49d68486\"]\n",
    "    sample_dict = dict(zip(FEATURE_COLUMNS, sample))\n",
    "    \n",
    "    for feature_name in CATEGORICAL_COLUMNS:\n",
    "        sample_dict[feature_name] = tf.expand_dims(sample_dict[feature_name], -1)\n",
    "        \n",
    "    for feature_name in CONTINUOUS_COLUMNS:\n",
    "        sample_dict[feature_name] = tf.expand_dims(tf.constant(sample_dict[feature_name], dtype=tf.int32), -1)\n",
    "    print(sample_dict)\n",
    "\n",
    "    return sample_dict\n",
    "\n",
    "m.predict(input_fn=pred_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export a model optimized for inference\n",
    "We can upload our trained model to the Cloud Machine Learning Engine's Prediction Service, which will take care of serving our model and scaling it. The code below exports our trained model to a `saved_model.pb` file and a `variables` folder where the trained weights are stored. \n",
    "\n",
    "The `export_savedmodel()` function expects a `serving_input_fn()`, which returns the mapping from the data that the Prediction Service passes in to the data that should be fed into the trained TensorFlow prediction graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.contrib.learn.python.learn.utils import input_fn_utils\n",
    "\n",
    "def column_to_dtype(column):\n",
    "    if column in CATEGORICAL_COLUMNS:\n",
    "        return tf.string\n",
    "    else:\n",
    "        return tf.float32\n",
    "\n",
    "def serving_input_fn():\n",
    "    feature_placeholders = {\n",
    "        column: tf.placeholder(column_to_dtype(column), [None])\n",
    "        for column in FEATURE_COLUMNS\n",
    "    }\n",
    "    # DNNCombinedLinearClassifier expects rank 2 Tensors, but inputs should be\n",
    "    # rank 1, so that we can provide scalars to the server\n",
    "    features = {\n",
    "        key: tf.expand_dims(tensor, -1)\n",
    "        for key, tensor in feature_placeholders.items()\n",
    "    }\n",
    "    \n",
    "    return input_fn_utils.InputFnOps(\n",
    "        features, # input into graph\n",
    "        None,\n",
    "        feature_placeholders # tensor input converted from request \n",
    "    )\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1373: get_timestamped_export_dir (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1378: get_temp_export_dir (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1388: get_input_alternatives (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1402: get_output_alternatives (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1412: build_all_signature_defs (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/utils/saved_model_export_utils.py:267: build_standardized_signature_def (from tensorflow.contrib.learn.python.learn.utils.saved_model_export_utils) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Switch to tf.estimator.Exporter and associated utilities.\n",
      "WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563697267/model.ckpt-2002\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: models/model_WIDE_AND_DEEP_1563697267/export/temp-1563704023/saved_model.pb\n",
      "model exported successfully to b'models/model_WIDE_AND_DEEP_1563697267/export/1563704023'\n"
     ]
    }
   ],
   "source": [
    "# Manually export\n",
    "export_folder = m.export_savedmodel(\n",
    "    export_dir_base = model_dir + '/export',\n",
    "    serving_input_fn=serving_input_fn\n",
    ")\n",
    "\n",
    "print('model exported successfully to {}'.format(export_folder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Experiments to manage the training workflow\n",
    "TensorFlow also offers an \"Experiments\" framework to help manage your training for you. Using it will take away some of the boilerplate and automate certain actions, but it's important to understand what is happening (what we did above) before using something like this.\n",
    "\n",
    "Notice that we don't need to create additional wrapper functions here, but rather, we can directly use the various functions we already defined previously. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.contrib.learn.python.learn import learn_runner\n",
    "from tensorflow.contrib.learn.python.learn.utils import saved_model_export_utils\n",
    "\n",
    "\n",
    "# get_model(model_type = 'WIDE_AND_DEEP', model_dir=model_dir)\n",
    "\n",
    "# output_dir is an arg passed in by the learn_runner.run() call.\n",
    "def experiment_fn(output_dir):\n",
    "    \n",
    "    print(output_dir)\n",
    "    \n",
    "    train_input_fn = generate_input_fn(train_file, BATCH_SIZE)\n",
    "    eval_input_fn = generate_input_fn(eval_file)\n",
    "    my_model = get_model(model_type=MODEL_TYPE, \n",
    "                  model_dir=output_dir)\n",
    "\n",
    "    experiment = tf.contrib.learn.Experiment(\n",
    "        my_model,\n",
    "        train_input_fn=train_input_fn,\n",
    "        eval_input_fn=eval_input_fn,\n",
    "        train_steps=1000\n",
    "        ,\n",
    "        export_strategies=[saved_model_export_utils.make_export_strategy(\n",
    "            serving_input_fn,\n",
    "            default_output_alternative_key=None,\n",
    "            exports_to_keep=1\n",
    "        )]\n",
    "    )\n",
    "    return experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models/model_WIDE_AND_DEEP_1563705602\n",
      "Model directory = models/model_WIDE_AND_DEEP_1563705602\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x138828358>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1563705602'}\n",
      "estimator built\n",
      "INFO:tensorflow:Skipping training since max_steps has already saved.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T10:46:52Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563705602/model.ckpt-1002\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-10:46:57\n",
      "INFO:tensorflow:Saving dict for global step 1002: accuracy = 0.759125, accuracy/baseline_label_mean = 0.252375, accuracy/threshold_0.500000_mean = 0.759125, auc = 0.7087525, auc_precision_recall = 0.45361945, global_step = 1002, labels/actual_label_mean = 0.252375, labels/prediction_mean = 0.30198872, loss = 0.52078867, precision/positive_threshold_0.500000_mean = 0.56143165, recall/positive_threshold_0.500000_mean = 0.2082219\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563705602/model.ckpt-1002\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: models/model_WIDE_AND_DEEP_1563705602/export/Servo/temp-1563706017/saved_model.pb\n",
      "CPU times: user 10.9 s, sys: 1.01 s, total: 11.9 s\n",
      "Wall time: 11.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# %%time 将会给出cell的代码运行一次所花费的时间，必须放在这个cell的第一行\n",
    "# manually train and eval\n",
    "exp = experiment_fn(model_dir)\n",
    "\n",
    "exp.train_and_evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models/model_WIDE_AND_DEEP_1563706169\n",
      "Model directory = models/model_WIDE_AND_DEEP_1563706169\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1391e8940>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1563706169'}\n",
      "estimator built\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T10:49:45Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563706169/model.ckpt-0\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-10:49:48\n",
      "INFO:tensorflow:Saving dict for global step 0: accuracy = 0.253725, accuracy/baseline_label_mean = 0.252375, accuracy/threshold_0.500000_mean = 0.253725, auc = 0.46673936, auc_precision_recall = 0.54403305, global_step = 0, labels/actual_label_mean = 0.252375, labels/prediction_mean = 0.96713245, loss = 276.68155, precision/positive_threshold_0.500000_mean = 0.25215778, recall/positive_threshold_0.500000_mean = 0.99554235\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Validation (step 1): loss = 276.68155, accuracy = 0.253725, labels/prediction_mean = 0.96713245, labels/actual_label_mean = 0.252375, accuracy/baseline_label_mean = 0.252375, auc = 0.46673936, auc_precision_recall = 0.54403305, accuracy/threshold_0.500000_mean = 0.253725, precision/positive_threshold_0.500000_mean = 0.25215778, recall/positive_threshold_0.500000_mean = 0.99554235, global_step = 0\n",
      "INFO:tensorflow:loss = 234.20192, step = 2\n",
      "INFO:tensorflow:Saving checkpoints for 102 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 10.3079\n",
      "INFO:tensorflow:loss = 0.45035392, step = 202 (3.549 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 204 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 92.6309\n",
      "INFO:tensorflow:Saving checkpoints for 306 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 91.8997\n",
      "INFO:tensorflow:loss = 0.53794026, step = 402 (2.244 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 408 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 72.3445\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T10:49:59Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563706169/model.ckpt-408\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-10:50:02\n",
      "INFO:tensorflow:Saving dict for global step 408: accuracy = 0.750725, accuracy/baseline_label_mean = 0.252375, accuracy/threshold_0.500000_mean = 0.750725, auc = 0.67955923, auc_precision_recall = 0.41523847, global_step = 408, labels/actual_label_mean = 0.252375, labels/prediction_mean = 0.2300231, loss = 0.52807736, precision/positive_threshold_0.500000_mean = 0.70666665, recall/positive_threshold_0.500000_mean = 0.021000495\n",
      "INFO:tensorflow:Validation (step 437): loss = 0.52807736, accuracy = 0.750725, labels/prediction_mean = 0.2300231, labels/actual_label_mean = 0.252375, accuracy/baseline_label_mean = 0.252375, auc = 0.67955923, auc_precision_recall = 0.41523847, accuracy/threshold_0.500000_mean = 0.750725, precision/positive_threshold_0.500000_mean = 0.70666665, recall/positive_threshold_0.500000_mean = 0.021000495, global_step = 408\n",
      "INFO:tensorflow:Saving checkpoints for 510 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 13.6443\n",
      "INFO:tensorflow:loss = 0.48375615, step = 602 (8.874 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 612 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 85.457\n",
      "INFO:tensorflow:Saving checkpoints for 714 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 88.1646\n",
      "INFO:tensorflow:loss = 0.51952714, step = 802 (2.286 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 816 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 64.6059\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T10:50:10Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563706169/model.ckpt-816\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-10:50:13\n",
      "INFO:tensorflow:Saving dict for global step 816: accuracy = 0.758775, accuracy/baseline_label_mean = 0.252375, accuracy/threshold_0.500000_mean = 0.758775, auc = 0.6935587, auc_precision_recall = 0.4353711, global_step = 816, labels/actual_label_mean = 0.252375, labels/prediction_mean = 0.26914382, loss = 0.51977175, precision/positive_threshold_0.500000_mean = 0.6269932, recall/positive_threshold_0.500000_mean = 0.10906389\n",
      "INFO:tensorflow:Validation (step 833): loss = 0.51977175, accuracy = 0.758775, labels/prediction_mean = 0.26914382, labels/actual_label_mean = 0.252375, accuracy/baseline_label_mean = 0.252375, auc = 0.6935587, auc_precision_recall = 0.4353711, accuracy/threshold_0.500000_mean = 0.758775, precision/positive_threshold_0.500000_mean = 0.6269932, recall/positive_threshold_0.500000_mean = 0.10906389, global_step = 816\n",
      "INFO:tensorflow:Saving checkpoints for 918 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 13.4877\n",
      "INFO:tensorflow:loss = 0.56702745, step = 1002 (9.226 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1002 into models/model_WIDE_AND_DEEP_1563706169/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.56702745.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-07-21T10:50:19Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563706169/model.ckpt-1002\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-07-21-10:50:22\n",
      "INFO:tensorflow:Saving dict for global step 1002: accuracy = 0.7585, accuracy/baseline_label_mean = 0.252375, accuracy/threshold_0.500000_mean = 0.7585, auc = 0.6968979, auc_precision_recall = 0.4378168, global_step = 1002, labels/actual_label_mean = 0.252375, labels/prediction_mean = 0.28664696, loss = 0.5207994, precision/positive_threshold_0.500000_mean = 0.5772098, recall/positive_threshold_0.500000_mean = 0.16106984\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1563706169/model.ckpt-1002\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: models/model_WIDE_AND_DEEP_1563706169/export/Servo/temp-1563706222/saved_model.pb\n",
      "Accuracy: 0.7584999799728394\n",
      "Model exported to [b'models/model_WIDE_AND_DEEP_1563706169/export/Servo/1563706222']\n"
     ]
    }
   ],
   "source": [
    "# Run the experiment\n",
    "\n",
    "model_dir=create_model_dir(model_type=MODEL_TYPE)\n",
    "metrics, output_folder = learn_runner.run(experiment_fn, model_dir)\n",
    "\n",
    "print('Accuracy: {}'.format(metrics['accuracy']))\n",
    "print('Model exported to {}'.format(output_folder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "In this Juypter notebook, we have configured, created, and evaluated a Wide & Deep machine learning model, that combines the powers of a Linear Classifier with a Deep Neural Network, using TensorFlow's Estimator and Experiment classes.\n",
    "\n",
    "With this working example in your toolbelt, you are ready to explore the wide (and deep) world of machine learning with TensorFlow! Some ideas to help you get going:\n",
    "* Change the features we used today. Which columns do you think are correlated and should be crossed? Which ones do you think are just adding noise and could be removed to clean up the model?\n",
    "* Swap in an entirely new dataset! There are many datasets available on the web, or use a dataset you possess! Check out https://archive.ics.uci.edu/ml to find your own dataset. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
